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Defining  SW  Rod  uct  Maturity 

•  US/ UK/ AUS  Software- intensive  Systems 
Acquisition  Working  Group  work  strand 

•  No  standard  definitions/  sc  ales 

•  Not  Softwa re  Tec  hnology  Readiness  levels  (TRL) 

-  Maturity  addresses  a  specific  product 

-  TRL  addresses  underlying  tec  hnology 

•  Highly  dependent  on  environment  and 
application  context 

•  Many  dimensions  of  maturity 
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The  Apptoac  h 


•  Gathered  a  group  of 
experts  to: 

-  Review  existing 
approaches 

-  Develop  characteristics 
and  infomnation  sources 

-  Develop  guidance  for 
source  selection 

-  Develop  RFQ/  RFP 
language 
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Focused  on  Source  Selection 


•  General  maturity  problem  is  extremely  difficult 

-  Limited  time  and  resources 

-  Need  for  significant  effort  to  work  on  development- 
based  maturity 

-  Some  promising  work  (MDA,  AF)  but  untried 

•  Source  selection  has  been  a  Congressional 
emphasis 

•  Source  selection  bounds  the  problem  to 
measuring  existing,  working  software  (e.g.  COTS* 
GOIS*  legacy) 
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Software  in  Source  Selection 


•  Many  different  kinds  of  source  selections 

-  Greenfield  vs.  Upgrade 

-  Traditional  business- process  IT system  implementation  vs. 
Command  and  Control  orembedded  software 

•  Different  kinds  of  software  in  programs 

-  Only  software  that  exists  has  determinable  maturity 

-  Aggregations  of  existentand  non-existent  software  are 
common 

•  Software  doesn’t  exist  •  Software  exists 

(Not  measurable)  (Measurable) 

-  Developmental  software  -  COTS 


-  GOTS 


-  Prototype 

-  NDI/ Legacy 

-  Experimental 


Observations 
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Software  product  maturity  is  value- 
neutral 

-  Mature  software  not  better  than 
immature  software  in  some 
instances;  must  be  interpreted  in 
light  of  risk 

•  Web-pages 

•  Proofk  of  concept 

Software  can  become  senile 

-  Number  of  changes  overwhelm  the 
architecture 

-  Environment  changes 

-  Utility  degrades 

level  of  understanding  of  context 
directly  impacts  risk  and 
interpretation  of  maturity 

-  Poorly  understood  application 
environment  or  la  rget  ma  kes  risk 
assessment  difficult 
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Utility  of  Software 


Notional  SW Maturity  Lifecycle 


TRL5  1RL8 
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Maturity  Evaluation  Characteristics 


•  Represent  a  teas/ 
dimensions  affecting 
product  maturity 

•  Must  be  considered 
both  separately  and  as 
a  group 

•  Weight  of  each 
characteristic  may 
differ  in  any  particular 
situation 

•  Must  be  evaluated 
against  intended 
purpose 
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Candidate  Characteristics 


1.  Understanding  of  the  potential  for  la  tent  defects 
within  the  product 

2.  Understanding  of  the  domain  of  product 
applicability 

3.  Predictability  of  product  behavior  (within  well- 
defined  parameters) 

4.  Product  stability 

5.  Product  supportability 

6.  Product  pedigree 
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Potential  for  latent  defects 


•  Addresses  the  risk  of  undetected  bugs 

•  Possible  measures  or  information  sources 


-  History  and  trends  of  types/frequency  of  faults 

-  Certifications  and  test  packages 
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Doma  in  of  pioduc t  a  pplic  a  bility 

•  Addresses  risk  of  suitability  of  the  product  to  the 
intended  task 

•  Possible  measures  or  information  sources 

-  Compatibility  measures 

-  Robustness  (adaptability,  scalability,  portability, 
security,  safety,  integrity,  etc.) 

-  Availability  and  quality  of  design  and  maintenance 
doc  uments 

-  Certifications  and  test  packages 

-  Specific  operational  environments) 

-  Limitations  on  product  use 
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Predictability  of  product  behavior 

•  Addresses  the  risks  associated  with  suitability  of 
operational  and  functional  quality 

•  Possible  measures  or  information  sources 

-  Test  regimen 

-  lest  coverage 

-  History  and  trends  of  types/frequency  of  faults 

-  MTBF 

-  Availability 

-  Recovery  from  faults 

-  Compatibility  measures 

-  Accuracy 

-  Completeness  of  features/  functions  definition 
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Productstability 


•  Addresses  the  risks 
associated  with  historic 
volatility  that  could  te- 
e  merge 

•  Possible  measures  or 
information  sources 

-  Release  history  and 
frequency 

-  History  and  trends  of 
types/  frequency  of  faults 

-  Obsolescence  potential 

-  Software  aging 
characteristics 
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Product  supportability 


*  Addresses  the  risks  associated  with  continuing  suitability 
of  the  product 

•  Possible  measures  or  information  sources 

-  Availability  of  training 

-  Availability  of vendoi? developer/ consultant  support 

-  Recovery  from  faults 

-  Mean  time  between  failure 

-  Availability  and  quality  of  design/  maintenance  documents 

-  Dependency  on  events  out  of  product  control 

-  Life  expectancy 

•  Hist  shipment  date 

•  End- of- life  plans 

•  Market  shane 

•  Maikettnend 

•  Rights  granted  on  discontinuation  of  product 
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Product  pedigree 


•  Addresses  the  risks  associated  with  the 
developed/  sources  for  the  product 

•  Possible  measures  or  information  sources 

-  Installed  base 

-  Market  share 

-  Market  trend 

-  Maturity  of  underlying  tec  hnology 

-  Customer  references 

-  Confidence  in  adherence  to 
standards 

-  History  of  upward  compatibility 
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Additional  factors 


•  Control  over  configuration/ evolution 

-  Does  the  acquisition  need  to  determine  when  or  how  the 
pro  due  twill  change  and  the  type  offeatures  that  may  be 
added  or  dropped? 

•  Predictability  of  evolution  and  obsolescence 

-  Does  the  acquisition  have  a  clear  understanding  of  the 
direction  and  speed  of  product  evolution  and  the  time 
remaining  in  the  products  likely  supported  life? 

•  Schedule 

-  Does  the  acquisition  understand  when  the  product  will  be 
available  or  updated  (such  as  availability  of  NDI  or  required 
prod  uc  t  func  tiona  lity )? 

•  Costs 

-  Does  the  acquisition  understand  the  full  costs  associated 
with  the  product;  such  as  licensing,  refresh,  maintenance 
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Additional  factors  -  2 


•  Architecture 

-  Will  the  product  require  significant  changes  to  an 
existing  software  architecture? 

•  Operational  Context 

-  Will  the  product  fit  the  current  or  envisioned  modes  of 
operation,  operational  environment  (e.g.  platform) 
and  process  context? 

•  Rtness  for  use 

-  Do  the  product  characteristics  meet  the  needs  of  the 
envisioned  use  (such  as  security,  availability,  and 
scalability)? 

•  Modification  of  product 

-  Will  there  need  to  be  modifications  to  the  productthat 
will  prevent  normal  developer/ vendor  refresh? 
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Additional  factors  -  3 


•  Release  synchronization 

-  Will  the  vendor  release  schedule  impact  operations? 

•  Pedigree  of  product  developer 

-  Does  the  acquisition  have  confidence  in  the 
developer/ vendor  (including  disclosure  of 
subcontractors)? 
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Context  impacts  risk 
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Additional  references 


•  ISO/ 1  EC  14598-4  Software  engineering  -  Product 
Evaluation  Part  4:  Process  for  acquirers 

-  extensive  guidance  on  evaluating  software  products. 

•  ISO/ 1  EC  9126- 1  Information  technology  - 
Software  quality  characteristics  and  metrics  - 
Parti :  Quality  c  ha  racteristic  sand 
subcharacteristics 

-  defines  software  quality  characteristics 

•  SB  Technical  Reports 

-  C  MU/  SB- 2004~TRr013  An  Alternative  to  Technology 
Readiness  Levels  for  Non- Developments  I  Software 

-  C  MU/  SB- 2003- TR- 023  Identifying  Commercial  Off-the- 
Shetf  (COTS)  Product  Risks:  The  COTS  Usage  Risk 
Evaluation. 
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Maturity  and  Agile  Development 
Approaches 

•  Agile  can  be  effective  determining  many  of  the 
characteristic  measures 

-  Probability  of  defects 

•  Test-driven  design 

•  Short  item  tions  yielding 
operational  functionality 

-  Domain  applicability 

•  More  involved  customer 

•  Acceptance  tests  for  each 
iteration 

-  Product  stability 

•  Automated  test  environments 

•  Continuous  integration 

-  Product  pedigree 

•  Nearly  all  agile  techniques 
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Summary 


•  Maturity  can  only  be  measured  on  existing 
software 

-  For  source  selection  this  means  COIS^  GOTS>  NDI, 
prototype,  experimental 

•  Initial  set  of  software  product  maturity 
c  ha  tacteristic  s  defined 

•  Maturity  evaluation  complex  -  dependent  on 
context  and  related  factors 

•  Agile  approaches  may  make  iteasierto 
determine  software  product  maturity 
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Questions? 
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